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Future  unmanned  systems  in  the  military  will  be  highly  heterogeneous  in  nature,  with 
vehicles  from  multiple  domains — aerial,  underwater,  and  land — working  in  collaborative 
teams  to  complete  a  variety  of  missions.  The  complexity  of  supervising  these  teams  will  be 
enormous  and  will  rely  on  human  creativity,  judgment,  and  experience.  Therefore,  the 
design  and  development  of  mission  planning  and  monitoring  technologies  must  be  rooted  in 
a  deep  understanding  of  the  human  operator’s  role  as  mission  manager,  and  must  effectively 
address  the  reasoning  skills  and  limitations  of  both  the  human  and  autonomous  intelligent 
system.  In  this  paper  we  present  our  work  to  approach  these  supervisory  issues  from  a 
human-centered  perspective.  We  first  review  the  findings  of  a  cognitive  task  analysis, 
through  which  we  defined  critical  informational  requirements  and  developed  display 
interfaces  for  human  operators  developing  and  executing  mission  plans  for  a  small  team  of 
underwater  and  aerial  unmanned  vehicles.  These  findings  raise  several  operations  issues  for 
unmanned  vehicle  management,  namely  (1)  vehicle  and  task  heterogeneity  and  (2)  the 
coordination  of  command  and  control  across  a  vehicle  team.  We  discuss  the  impact  of  both 
of  these  design  requirements  on  the  human-centered  development  of  mission  planning  tools 
for  unmanned  systems.  Finally,  we  introduce  an  investigative  approach  to  support  the  rapid 
evaluation  of  interfaces  that  flexibly  accommodate  alternative  command  and  control 
philosophies  for  heterogeneous  automated  systems  using  a  combination  of  modeling  and 
human-in-the-loop  evaluation  processes 


I.  Introduction 

Future  naval  operations  in  the  littoral  environment  are  expected  to  make  extensive  use  of  coordinated  unmanned 
vehicle  teams  to  support  a  range  of  operations,  including  intelligence,  surveillance,  and  reconnaissance  (ISR),  as 
well  as  anti-terrorism/force  protection  (AT/FP),  suppression  of  enemy  air  defenses  (SEAD),  and  anti-mine  and  anti¬ 
submarine  warfare  (AMW,  ASW).  '' 2  While  great  strides  are  being  made  in  the  development  of  unmanned  aerial, 
undersea  and  surface  vehicles  (collectively  UVs),  the  development  of  advanced  operational  capabilities  that  will 
effectively  support  the  autonomous  operation  of  heterogeneous  teams  of  these  vehicles  with  minimal  human 
intervention  is  proceeding  more  slowly.  Within  the  scope  of  the  Intelligent  Autonomy  program,  the  Office  of  Naval 
Research  (ONR)  is  pursuing  technological  advancements  to  support  future  operations  in  the  littoral  environment  by 
developing  advanced  autonomous  mission  planning  and  execution  technologies  for  heterogeneous  teams  of  UVs. 
One  significant  goal  of  the  IA  program  is  to  maximize  the  vehicle  to  operator  ratio  of  such  heterogeneous  UV  teams 
to  effectively  transition  the  human  operator’s  role  from  an  in-the-loop  controller  to  a  supervisory  manager,  or  on- 
the-loop  controller. 

While  many  future  unmanned  systems  will  reduce  the  extent  to  which  human  operators  engage  in  direct, 
manual  control  of  vehicle  behaviors,  humans  will  still  be  deeply  involved  in  planning,  higher-level  operation,  and 
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contingency  intervention  (including  coping  with  system  failures).  Therefore,  any  system  of  so-called  “unmanned” 
vehicles  will  remain  a  manned  system.  This  concept  of  coordinated  control  between  human  operators  and  automated 
platforms  is  described  as  a  mixed-initiative  team.  Within  mixed-initiative  teams,  the  transition  of  the  human’s  role 
from  manual  to  supervisory  control  introduces  a  new  layer  of  cognitive  complexity,  which  stems  from  the  need  for 
effective  communication  between  human  operators  and  multiple  autonomous  mission  planning  and  execution 
systems.  Mission  planning  for  mixed-initiative  teams — both  prior  to  and  during  actual  execution — is  an  inherently 
human  process  that  can  be  aided  by  automation  but  must  always  depend  on  human  strengths  of  creativity,  judgment, 
and  experience,  while  guarding  against  decision  biases  and  error.  Within  future  mixed-initiative  systems,  operators 
will  be  expected  to  manage  multiple  tasks  at  once,  such  as  mission  planning/replanning  and  mission  monitoring,  for 
vehicles  in  the  air,  under  and  on  the  water,  and  on  land.  However,  increases  in  the  number  of  available  information 
sources,  the  volume  of  incoming  information,  and  operational  tempo — all  of  which  contribute  to  the  overall 
cognitive  demand  placed  on  operators — could  become  limiting  constraints  to  the  success  of  these  network-centric 
processes.3  Therefore,  the  design  and  development  of  technologies  supporting  the  supervisory  control  of  unmanned 
vehicle  teams  must  be  rooted  in  an  understanding  of  and  respect  for  the  human’s  role  as  mission  manager.  As  a  wide 
variety  of  highly-automated  vehicle  platforms  and  sensor  packages  become  available,  designers  will  need  to  address 
the  new  functionality  these  systems  afford  through  a  human  factors-based  process  that  maximizes  human 
capabilities  when  leveraging  these  resources. 

In  particular,  two  major  concerns  regarding  the  development  of  interfaces  for  human  supervision  of  unmanned 
vehicle  teams  are  explored  in  this  paper:  (1)  the  heterogeneity  of  assets  within  the  unmanned  team,  and  (2)  the 
allocation  of  heterogeneous  team  assets  and  task  responsibilities  amongst  multiple  operators. 

With  respect  to  heterogeneity,  the  US  Navy  is  targeting  the  deployment  of  numerous  types  of  unmanned 
vehicles  within  the  future  littoral  environment.  In  this  example,  heterogeneity  not  only  refers  to  the  simultaneous 
deployment  of  aerial,  surface,  underwater,  and  ground  vehicles  that  operate  in  different  physical  domains,  but  also 
implies  the  integration  of  unmanned  assets  with  different  operational  capabilities,  sensor  packages,  and  levels  of 
intelligent  automation.  To  successfully  address  these  types  of  heterogeneity,  it  will  be  crucial  to  avoid  typical 
system  engineering  approaches  in  which  human-machine  interaction  is  designed  for  very  specific  and  isolated 
vehicle  capabilities.  Such  development  processes  result  in  stovepiped  systems  that  cannot  be  easily  integrated  to 
support  long-term  interests  for  coordinated  mission  operations  in  multiple  domains  or  across  multiple  platforms. 
Rather,  we  believe  that  success  lies  in  providing  unified  human/machine  interfaces  that  support  an  operator’s 
seamless  interaction  with  the  full  range  of  targeted  vehicles  and  tasks  and  are  carefully  targeted  to  minimize  the 
cognitive  costs  of  rapidly  switching  between  these.  Only  by  accommodating  this  seamless  integration  in  the  initial 
design  process  and  then  layering  the  specific  vehicle  interaction  requirements  on  top  of  that  design,  will  it  be 
possible  to  support  the  ultimate  goal  of  integrated  mission  operations. 

Most  designs  for  controlling  unmanned  vehicles  reflect  an  assumption  that  deployment  will  occur  in  an  isolated 
operational  environment  where  a  single  operator  (or  even  small  group  of  collocated  operators)  will  be  independently 
responsible  for  managing  a  team  of  independently  “owned”  vehicles.  However,  it  is  more  likely  that  large-scale 
missions  will  require  the  deployment  of  multiple  teams  of  unmanned  assets  from  multiple  distributed  command  and 
control  platforms,  and  mission  management  will  be  a  coordinated  and  collaborative  process  across  these  integrated 
human  and  semi-autonomous  teams.  To  address  this  issue  of  coordination,  our  strategy  must  accommodate  the 
notion  of  multi-platform/multi-asset  allocation  into  the  human  factors-based  design  process  from  the  start. 
Unfortunately,  there  is  a  lack  of  clarity  throughout  the  unmanned  systems  community  about  the  exact  nature  of  such 
strategies  for  supervisory  allocation.  Therefore,  it  is  critical  that  any  approach  to  enable  a  flexible  design  and 
development  process  support  examination  and  evaluation  of  a  range  of  approaches  to  command  and  control 
allocation  across  multiple  operator  teams. 

In  this  paper,  we  present  the  initial  results  of  work  undertaken  to  address  these  issues  from  a  human-centered 
perspective.  We  first  review  the  results  of  a  preliminary  analysis  and  design  effort  aimed  at  developing  effective  and 
efficient  displays  for  supervising  simple  unmanned  vehicle  team  within  a  littoral  combat  environment.  This  example 
shows  how  informational  requirements  critical  to  supporting  human  operators  in  the  development  and  execution  of 
mission  plans  may  be  defined  for  a  particular  mixed-initiative  team  and  illustrates  how  these  informational 
requirements  can  be  leveraged  in  the  design  of  display  concepts  to  support  these  informational  requirements  in  an 
efficient  and  effective  manner  with  respect  to  the  mission  context.  The  findings  from  this  initial  analysis  and  design 
effort  point  to  the  importance  of  several  outstanding  questions  regarding  issues  of  collaboration  and  control 
allocation  in  the  management  of  heterogeneous  unmanned  vehicles.  We  discuss  the  impact  of  these  issues  on  the 
development  of  mission  planning  tools  for  unmanned  systems  from  a  human-centered  perspective.  Finally,  we 
introduce  an  investigative,  simulation-based  approach  for  the  rapid  evaluation  of  multiple  candidate  interface 
designs  that  accommodate  alternative  command  and  control  philosophies  for  heterogeneous  automated  systems.  We 
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are  currently  using  this  approach  to  perform  human-centered  design  and  development  that  targets  the  human 
operator’s  ability  to  interact  collaboratively  with  both  unmanned  systems  and  other  distributed  human  operators. 

II.  Developing  Mission  Planning  and  Monitoring  Interface  for  an  Unmanned  Vehicle  Team 

Approaches  to  mission  management  for  mixed-initiative  operations  typically  center  upon  visual  interfaces 
designed  to  support  a  single  human  operator  controlling  a  single  vehicle.  Future  unmanned  operations,  however,  will 
require  supervisory  interfaces  that  allow  a  single  operator  to  manage  a  team  of  multiple  vehicles  working  to 
complete  one  or  more  tasks  simultaneously.  Here,  we  describe  a  preliminary  effort  to  design  a  human-centered 
display  interface  for  managing  a  simple  team  of  unmanned  vehicles  in  a  limited  operational  context.  We  first 
describe  the  analytical  methods  employed  to  determine  the  informational  and  task-related  requirements  such  an 
interface  would  need  to  support,  and  then  present  an  overview  of  the  design  implementations  used  to  address  these 
requirements. 

A.  Identifying  Information  Requirements  through  a  Hybrid  Cognitive  Task  Analysis 

One  significant  hurdle  to  the  development  of  interfaces  for  novel  or  futuristic  systems  is  establishing  a  sufficient 
understanding  of  how  those  interfaces  will  be  used  in  the  fulfillment  of  particular  operational  demands,  including 
who  will  use  them,  what  context  they  will  be  used  in,  and  what  specific  tasks  they  will  support.  Various  approaches 
to  cognitive  tasks  analysis  (CTA)4  have  been  proposed  and  demonstrated  for  deriving  information  requirements  and 
interface  design  concepts  for  systems  that  already  exist  in  some  form.  However,  most  established  CTA  techniques 
are  difficult  to  apply  in  the  design  of  novel,  futuristic  systems  for  which  no  subject  matter  experts,  documentation,  or 
previous  system  implementations  exist.5, 6  In  our  design  effort,  the  goal  was  to  develop  interfaces  to  support  a  single 
operator  controlling  a  team  of  multiple  unmanned  vehicles — something  infeasible  with  existing  technical 
capabilities,  limiting  the  effectiveness  of  traditional  CTA  approaches.  To  address  these  limitations,  a  multi-tiered 
approach  to  CTA  was  developed  and  used,  drawing  upon  several  distinct  analytical  methods.  Details  concerning  the 
specific  methods  of  this  “hybrid”  CTA  approach  are  described  by  Ref.  7.  Here,  we  summarize  the  previously- 
reported  application  of  these  techniques  to  the  current  design  problem  and  provide  an  overview  of  the  resultant 
analytical  outputs. 

Prior  to  executing  the  hybrid  CTA  process,  we  developed  a  detailed  description  of  an  operational  activity 
scenario  describing  operator  roles  and  unmanned  vehicle  tasks  and  behaviors  in  completing  a  particular  mission. 
This  scenario  described  the  manner  in  which  a  supervisory  interface  would  be  leveraged  to  plan,  monitor,  and  adapt 
unmanned  vehicle  behaviors  in  the  fulfillment  of  a  particular  combat  mission,  and  was  used  to  ground  the  generation 
of  information  requirements  through  the  hybrid  CTA  process.  While  futuristic  in  nature,  the  operational  scenario 
was  intended  to  be  realistic  in  scope  and  sufficiently  detailed  to  clearly  identify  the  supervisory  operator’s 
anticipated  responsibilities  with  respect  to  tasking  the  system  of  unmanned  vehicles,  understanding  proposed 
mission  plans  to  achieve  that  tasking,  and  monitoring  the  mission  execution  against  those  generated  plans.  Briefly, 
this  scenario  involved  a  single  human  operator’s  supervision  of  a  team  of  unmanned  underwater  vehicles  (UUVs) 
performing  clandestine  ISR  operations  within  an  enemy  harbor.  The  unmanned  vehicle  team  was  comprised  of  two 
UUVs  responsible  for  entering  the  harbor,  obtaining  surface  imagery,  and  tracking  targeted  vessels,  as  well  as  two 
UUVs  that  served  as  sentries  at  the  harbor’s  mouth.  In  addition,  communications  between  the  UUVs  and  the  human 
operator  were  supported  by  an  unmanned  aerial  vehicle  (UAV)  on  a  shared  network.  Although  the  scenario  was 
fairly  limited  in  scope,  it  did  exhibit  some  key  qualities  of  heterogeneity  anticipated  for  future  unmanned  operations, 
including  multiple  tasks  (imaging,  tracking,  communications  support)  and  vehicle  domains  (underwater,  air). 

Based  on  this  limited  operational  scenario,  a  hybrid  CTA  was  performed  to  generate  the  functional  and  display 
requirements  for  a  supervisory  interface  that  would  allow  a  single  operator  to  coordinate  the  control  of  four  UUVs 
via  a  UAV  communications  link.  The  hybrid  CTA  process  began  with  the  definition  of  a  Scenario  Task  Overview, 
through  which  the  operational  scenario  described  above  was  first  separated  into  distinct  phases,  with  boundaries 
marked  by  changes  in  the  operator’s  anticipated  goals,  tasks,  and  behaviors.  These  individual  phases  were  then 
further  decomposed,  with  anticipated  sub-goals  and  sub-tasks  enumerated  using  a  hierarchical  structure.  Through 
this  process,  three  primary  mission  phases  were  identified  and  described:  (1)  mission  planning,  (2)  mission 
execution,  and  (3)  vehicle  recovery.  Within  these  phases,  a  total  of  25  distinct  mission  sub-tasks  were  identified  and 
described  in  detail.8  A  partial  example  of  the  Scenario  Task  Overview — a  description  of  the  mission  execution  phase 
of  the  operational  scenario — is  provided  in  Table  1. 
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Table  1.  Scenario  Task  Overview  for  Mission  Execution  Phase. 


UUV  Team  Harbor  ISR  Scenario 

Phase  2:  Mission  Execution 

Phase  Goals 

Phase  Breakdown 

1.  Launch  UUVs 

•  Launch  Search  UUVs 

•  Launch  Sentry  UUVs 

2.  Acquire  Target 

•  LCS  tracks  progress  of  UUVs,  making  sure  UUVs  are  at  expected  positions  at  scheduled  checkpoints 
(geo-fix  should  update  actual  location) 

•  LCS  tracks  status  &  availability  of  UAV  (ongoing) 

•  Search  UUVs  surface  to  scan  shoreline  at  planned  location/time  (updated  geo-fix  position  data  sent 
via  UAV,  if  UAV  is  in  range) 

•  If  ATR  flags  an  image  during  a  Search  UUV  scan,  EO  imagery  is  sent  to  LCS  via  UAV.  If  UAV 
unavailable,  UUV  loiters,  resurfacing  and  resending  at  next  scheduled  checkpoint.  Continue  until 

UAV  is  available.  Once  EO  imagery  is  sent,  UUV  loiters  until  Acknowledgement  (Ack)  is  received 
from  LCS  -  surfacing  at  each  scheduled  checkpoint. 

•  LCS  examines  EO-imagery,  then  sends  Ack  back  to  all  UUVs.  If  Ack  is  positive,  UUV  continues  to 
loiter  in  current  position,  monitoring  confirmed  target.  If  Ack  is  negative,  UUV  resumes  search  task. 

•  Phase  is  complete  once  one  of  the  UUVs  has  received  a  positive  Ack  from  the  LCS.  Second  Search 
UUV  should  be  retasked  to  confirmed  target  location  for  redundant  monitoring  (or  retasked  to  another 
potential  target  or  returned  to  LCS) 

3.  Monitor  Target 

•  Search  UUV  should  continue  to  surface  at  scheduled  checkpoints  to  continue  monitoring  target  -  ATR 
should  continue  to  flag  target  as  contact  of  interest  and  update  imagery. 

•  When  onboard  ATR  no  longer  has  target  in  camera  range,  target  tracking  should  be  handed  off  to 

UAV.  Last  known  location  should  be  available  to  Sentry  UUVs,  to  UAV’s  MTI  program,  and  to  LCS. 

•  When  target  is  lost  by  Search  UUV(s),  Sentry  UUVs,  UAV  (if  available)  and  LCS  should  be  given  the 
last  known  location  by  the  UUV  as  well  as  historical  and  predicted  track  of  target. 

•  Phase  is  complete  once  UAV  is  tracking  the  target,  or  Search  UUVs  handoff  to  Sentry  UUVs. 

•  LCS  should  retask  or  recall  Search  UUVs 

4.  Track  Target 

•  UAV  should  use  last  known  location  from  UUV  +  MTI  software  to  track  target.  UAV  should  send  LCS 
MTI  feed. 

•  LCS  should  monitor  LCS  MTI  feed. 

•  LCS  should  determine  estimated  time  of  arrival  of  target  at  harbor  entrance  based  on  MTI  feed  from 
UAV  and  schedule  Sentry  UUVs  to  surface  and  capture  EO-imagery  of  expected  target  location  at 
that  time  via  UAV  communication  link  (this  should  be  automated  to  the  highest  degree  possible). 

•  Phase  is  complete  once  target  reaches  Sentry  UUVs.  UAV  tracking  could  be  discontinued  at  this  time. 

5.  Exit  Harbor 

•  Within  a  predetermined  window  of  time,  Sentry  UUVs  should  surface  and  wait  for  target  arrival. 

•  UUV  should  capture  and  send  collected  EO-imagery  to  LCS  via  UAV.  If  UAV  is  unavailable,  surface 
at  scheduled  time  intervals  to  retry  EO-imagery  transmission.  Continue  cycle  until  Ack  is  received 
from  LCS.  Regardless  of  Ack  status,  at  least  1  UUV  will  track  the  target  out  of  the  harbor  based  on  a 
set  of  predetermined  criteria. 

•  LCS  should  determine  tracking  profile  of  Sentry  UUVs  and  determine  when  they  will  be  retasked  or 
recalled. 

Following  the  decomposition  of  tasks  within  the  Scenario  Task  Overview,  an  Event  Flow  Diagram  was  created 
to  analyze  and  capture  the  temporal  constraints  of  the  selected  mission  scenario.  These  temporal  constraints  govern 
when  particular  events  or  control  activities  must  occur  within  the  context  of  the  overall  mission,  particularly  in 
relation  to  other  events.  An  example  of  an  overview-level  Event  Flow  Diagram  for  the  entire  mission  scenario  is 
shown  in  Figure  1,  on  the  following  page.  The  Event  Flow  Diagram  depicts  three  basic  classes  of  events  relevant  to 
the  supervisory  control  of  unmanned  vehicles:  loops,  decisions,  and  processes.  Loops  (L)  represent  automated 
processes  that  will  potentially  continue  in  an  iterative  fashion  until  some  pre-determined  event  occurs.  Decisions  (D) 
represent  events  that  require  some  type  of  knowledge-based  input  from  the  human  supervisor  before  a  particular 
vehicle  behavior  continues  or  changes.  Processes  (P)  are  situations  requiring  some  form  of  human-computer 
interaction  beyond  simple  decision-making  to  support  a  mission  subtask.  In  this  example,  these  event  classes  are 
labeled  for  cross-reference  with  subsequent  products  of  the  hybrid  CTA  process. 
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MISSION 
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MISSION 

PLANNING 

EXECUTION 

RECOVERY 

Figure  1.  Event  Flow  Diagram  for  UUV  control  during  the  mission  scenario 
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Drawing  on  the  temporal  description  of  the  mission  phase  sub-tasks,  a  Situation  Awareness  (SA)  Requirements 
analysis  was  performed.  This  analysis  used  traditional  methods  to  generate  SA  requirements  for  each  of  the  three  SA 
levels:  perception,  comprehension,  and  projection.9  For  each  of  these  levels,  the  analytical  team  specified  the 
situation  awareness  requirements  for  each  of  the  mission  phases  and  associated  sub-tasks  identified  by  the  Scenario 
Task  Overview.  A  partial  example  of  output  of  this  process  is  shown  in  Table  2. 

Table  2.  Scenario  Task  Overview  for  Mission  Execution  Phase. 


UUV  Team  Harbor  ISR  Scenario 

Phase  2:  Mission  Execution  -  Acquire  Target 

Level  1 :  Perception 

Level  2:  Comprehension 

Level  3:  Projection 

•  Visual  and  audible  alert  when 

UAV  leaves  or  returns  to  on- 
station  duty  (D2) 

•  Error/alert  message  clarification  (LI , 

D2,  PI,  D3) 

•  Schedule  of  estimated  UAV  on-station 
availability  should  be  provided  on  a 
visual  timeline  (D2,  D3) 

•  Uncertainty  of  estimated  timeframes 
should  be  indicated  on  availability 
timeline  (D2,  D3) 

•  Potential  missed  communications 
points  (D2,  D3) 

•  Future  likely  UUV  tracks  (D2,  D3) 

•  Vehicle  limitations  (when  predicted  to 
exceed  some  safe  region)  (D3) 

•  UUV  schedules  (D2,  D3) 

•  Prediction  of  system  health/status  (D2) 

•  All  agents’  position  information 
(D3) 

•  Hazardous  areas  (LI,  D2) 

•  Geo-spatial  boundaries  (LI,  D2) 

•  Vehicle  limitations  (on  demand)  (PI, 

D3) 

•  Indicate  communications  link 
coveraqe  range  when  on-station 
(D2,  D3) 

•  Sensor  coverage  should  be 
visualized  on  tactical  map  (D2, 

D3) 

•  UUV  schedules  (D2,  D3) 

.  Health  and  status  of  UUVs  (LI ,  D2) 

•  Visual/audio  feedback  for 

confirmation  of  target  acquisition 
(D2,  L1,D3) 

•  Strength  of  comms  link  to  UUV 
scheduled  to  check  in  should  be 
indicated  on  tactical  map  (based  on 
current  position  of  UAV  and  comms 
range)  (D2,  LI,  D3) 

•  Expected  connection  should  be 
indicated  at  UUV  scheduled 
checkpoint  time  -  if  UAV  out  of  range 
/  unavailable,  missed  connection 
should  be  indicated  (LI,  D2) 

•  Temporal  constraints  (PI) 

Finally,  a  Control  Task  Analysis10  was  performed  to  analyze  potential  behavior  of  the  human  operator  at  each  of 
the  decision  processes  outlined  by  the  Event  Flow  Diagram.  Separate  Decision  Ladders11  were  used  to  capture  the 
states  of  knowledge  and  information  processes  required  by  the  human  operator  to  perform  these  component 
decisions.  These  decision  ladders  were  then  augmented  with  design  concepts  describing  how  the  steps  within  each 
ladder  could  be  supported  by  an  information  display,  as  well  as  alternative  automation  implementation  strategies.  An 
example  of  one  such  augmented  decision  ladder  from  this  analysis  is  shown  in  Figure  2,  on  the  following  page. 
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ATR  Acknowledgement 
Decision  Ladder 

Target  of  interest  intel  (e.g.,  catalogue  I 
of  previous  images,  sonar  tracks,  etc.)  | 
should  be  available  simultaneously  with 
EO-imaqerv  for  direct  comparison. 


INTERPRET:  Decide  whether 
EO-imagery  contains  target  of 
interest 

- 1  - 


Examine  target  of  interest  & 
compare  to  other  intel  (images  or 
satellite  imagery  of  target  of 
interest,  ship  class,  ship 
schematics,  ship  name, 
distinguishing  features  of  target  of 
interest.) 


Parameters  for  imagery 
change  requests  should  be 
available  (e.g.,  sea  state, 
weather,  camera  angle) 


Current  UAV  availability  and 
comms  link  (health/status) 


Request  further  details 
of  target  of  interest 
imagery 


SEND  UUV  Acknowledgement 
('positive'/ 'negative')  via  UAV 
comm  link 


IDENTIFY  if  EO-imagery  quality  is 
sufficient  for  target  ID  assessment 


EO-imagery  (browsing  and 
zooming),  situation  map 
shows:  which  UUV  sent 
imagery,  position  of  UUV, 
camera  angle  &  camera 
range  during  image  capture 
&  ATR  classification. 


Display  confirmation  UUV  has 
been  notified  ofAck  message. 
UUV  position  should  be  updated 
on  display,  expected  surface 
schedule  should  be  displayed. 


Perceive  ATR  image  (received 
EO-imagery,  relative  location  of 
UUV  and  imagery,  ATR 


1 

L  / 

Visual  alert  and 

audible  alert, 
i _ 

ACTIVATION:  ATR  image  received  from 
Search  UUV  (visual  or  audible  alert) 

— 

EXECUTE:  UUV/ UAV  is  monitoring 
target 


Figure  2.  Example  of  a  Decision  Ladder  for  the  ATR  Acknowledgment  decision  (D3)  within  the  Acquire 
Target  sub-phase  of  Mission  Execution. 


B.  Transforming  Information  Requirements  into  Interface  Design  Concepts 

Based  on  the  outputs  of  the  hybrid  CTA  described  above,  a  detailed  HCI  design  was  developed  to  support  a 
single  operator  supervising  control  of  the  unmanned  vehicle  team  depicted  in  the  mission  scenario.  This  supervisory 
interface  was  designed  to  provide  a  suite  of  advanced  visualization  tools  that  increase  the  operator’s  ability  to 
intuitively  understand  and  manipulate  the  scheduled  activities,  or  plans,  of  multiple  unmanned  vehicles  involved  in  a 
mission  by  addressing  the  information  requirements  and  decision-making  processes  outlined  through  the  hybrid 
CTA.  The  interface  consists  of  three  primary  displays,  presented  in  parallel  across  multiple  display  monitors,  as 
shown  in  Figure  3,  on  the  following  page.  These  displays  are:  (1)  a  Map  Display,  (2)  a  Health  and  Status  Display, 
and  (3)  a  Task  Display.  The  content  and  structure  of  the  information  provided  to  the  operator  through  each  of  the 
three  unique  displays  was  governed  by  the  information  requirements  and  task  activity  insight  generated  by  the 
hybrid  CTA  process,  while  the  graphical  mechanisms  used  to  present  this  information  was  informed  by  human- 
centered  design  principles.  In  particular,  informational  resources  were  distributed  across  multiple  displays  in  an 
attempt  to  allocate  common  supervisory  control  activities  and  to  preserve  operator  SA  by  reducing  the  extent  to 
which  discrete  tasking  operations  obscure  the  availability  of  continually  evolving  mission  data  and  world  state 
information. 
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Map  Display  Health  and  Status  Display  Task  Display 

Figure  3.  The  three  component  displays  developed  to  support  supervisory  control  of  the  notional  unmanned 
vehicle  team  described  by  the  operational  scenario. 

The  interface’s  Map  Display  (Figure  4)  was  designed  to  marshal  geospatial  information  relevant  to  the  entire 
mission  into  a  single  display.  This  display  allows  the  operator  to  see  the  positions  of  the  UAV  and  UUVs — as  well 
as  the  positions  of  other  known  or  estimated  entities,  including  blue,  red,  and  neutral  vehicles — in  relation  to  the 
operational  area  and  the  command  and  control  platform.  At  the  operator’s  discretion,  the  Map  Display  can  also 
depict  specially  defined  geospatial  regions  such  as  “no-swim”  or  “no-communication”  zones,  planned  and  traversed 
vehicle  routes,  current  and  predicted  target  locations,  or  other  mission-relevant  elements  with  spatial  properties. 
These  additional  layers  can  be  toggled  using  a  display  filter  menu,  which  allows  the  operator  to  quickly  and  easily 
create  customized  display  interfaces  that  depict  the  most  relevant  and  useful  information — or  hide  the  most 
extraneous  or  distracting  information  — for  supporting  awareness,  reasoning,  and  planning  during  the  evolving 
mission.  Because  unmanned  vehicles  may  be  operating  in  many  domains  simultaneously  for  a  given  mission  (e.g. 
water  surface,  underwater,  air,  land)  there  will  be  a  need  for  the  operator  to  flexibly  examine  the  information  most 
relevant  to  the  particular  vehicle  or  task  that  is  being  attended  to  at  any  given  time.  For  example,  when  the  operator 
is  planning  or  evaluating  potential  UUV  routes,  local  bathymetry  data  can  be  added  to  the  Map  Display.  This 
bathymetry  data  can  then  be  removed  to  reduce  clutter  when  the  operator  is  interacting  with  a  UAV  asset,  at  which 
time  it  may  be  more  relevant  to  display  topological  data,  or  anti-aircraft  threat  zones. 
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The  Health  and  Status  Display  (Figure  5)  provides  an  overview  of  the  situational  information  most  critical  to 
the  operator’s  supervisory  control  of  the  mixed-initiative  team.  This  includes  relevant  operating  parameters  from 
individual  vehicles,  such  as  internal  temperature,  oil  pressure,  and  remaining  fuel  level.  Also  depicted  is  the 
communication  status  between  the  vehicle  team  and  the  command  and  control  platform.  In  this  case,  a  network 
diagram  depicts  communications  from  each  UUV  being  routed  through  the  single  UAV  to  the  base  of  operations 
aboard  a  Littoral  Combat  Ship  (LCS).  The  diagram  is  augmented  to  depict  the  strength  of  the  wireless  connection 
between  each  vehicle,  allowing  the  operator  to  quickly  ascertain  the  overall  status  of  communication  between  the 
entire  team  while  performing  monitoring,  troubleshooting,  or  re-planning  activities. 

At  the  top  of  the  display,  a  plan  timeline  allows  the  operator  to  see  the  linked,  temporal  constraints  of  future  and 
executed  mission  plans  at  a  glance,  while  providing  drill-down  access  to  detailed  timelines  for  each  vehicle.  The 
timeline  is  augmented  to  display  additional  mission  data  in  a  number  of  ways.  For  example,  color-coded  regions 
indicate  UUV  status  (e.g.,  dark  blue  for  submerged,  light  blue  for  surfaced);  bounded  areas  indicate  times  during  the 
mission  when  the  UAV  will  be  able  to  serve  as  a  communications  link  for  ISR  download,  decision 
acknowledgement,  or  re-tasking;  icons  show  planned  or  completed  task  activities,  such  as  ISR  image  collection. 
Finally,  in  addition  to  providing  overall  health  and  temporal  status  indications,  all  major  alarms — such  as  low 
vehicle  fuel  levels,  unexpectedly  lost  communications  links  and  missed  mission  deadlines — are  also  routed  through 
this  main  display  window. 


Figure  5.  Health  and  Status  Display. 

The  Task  Display,  (Figure  6)  collates  detailed,  fine-grained  information  for  decision-making  and  task-level 
vehicular  control  in  a  single  location,  separate  from  the  Map  and  Health  and  Status  displays.  The  Task  Display 
allows  the  operator  to  make  decisions  based  on  information  transmitted  by  the  vehicles’  sensor  packages  and  control 
the  current  and  future  actions  of  each  unmanned  vehicle.  For  example,  it  displays  ISR  images  received  from 
individual  UUVs  and  supports  the  operator  in  reviewing  and  performing  related  mission  actions  on  those  images 
(such  as  verifying  that  a  UUV  has  in  fact  located  a  particular  target,  or  requesting  additional  EO  imaging).  This 
multifaceted,  user-configurable  interface  also  allows  the  operator  to  collaborate  with  other  mission  operators  or  the 
commander,  and  to  define  or  update  mission  tasks  and  vehicle  behavior  parameters  (e.g.,  minimum  approach 
distances  for  obstacles  or  targets,  maximum  wait  time  before  surfacing  for  radio  check-in). 
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Figure  6.  Task  Display. 


One  key  component  of  the  Task  Display  is  the  Dynamic  Contextual  Decision  Tree  control  interface  (DCDT)12. 
This  interactive  display  component  (Figure  7)  dynamically  adapts  the  supervisory  control  options  available  to  the 
operator  to  address  the  context  of  particidar  mission  tasks,  and  presents  the  operator  with  two  types  of  information: 
(1)  actions  reflecting  a  supervisory  decision  to  be  made  by  the  operator,  and  (2)  actionable  information  sources 
which  may  support  the  operator  in  better  making  this  required  decision.  For  example,  in  this  particular  mission 
scenario,  the  UUV  automated  target  recognition  (ATR)  system  captures  an  image  of  a  vessel  within  the  harbor  and 
identifies  the  vessel  as  the  SS  Windsor.  Mission  requirements  necessitate  that  the  human  supervisory  controller 
verify  all  target  recognition  tasks.  In  this  case,  the  DCDT  interface  provides  image  information  to  the  operator  (e.g. 
“Possible  Match:  SS  Windsor;  Confidence:  Fligh”)  and  then  gives  the  operator  a  choice  of  actions  regarding  the 
confirmation  of  this  targeting  activity  (e.g.  “Confirm  as  SS  Windsor  with  High  confidence”,  “Confirm  as  SS 
Windsor  with  Medium  Confidence”,  etc.),  or  context-sensitive  actions  that  will  help  them  make  a  confirmation  (e.g. 
“Look  up  SS  Windsor  in  ship  images  database”).  The  DCDT  is  dynamically  branched  to  reflect  subsequent  actions 
by  the  operator.  For  example,  selecting  the  control  action  “Negate  match  and  provide  alternative”  results  in  new 
“Choose  from...”  lists,  including  alternative  control  actions  (e.g.  “Confirm  as  SS  Vincence”,  “Confirm  as  Other”)  or 
a  pruned  alternative  action  list  (e.g  the  “Look  up  SS  Windsor”  options  are  removed  from  the  list ). 


10 

American  Institute  of  Aeronautics  and  Astronautics 


Image  Information 


Possible  Match: 

SS  Windsor 

Match  Confidence: 

High 


Choose  from  . . . 

Confirm  as  SS  Windsor  with  High  confidence 
Confirm  as  SS  Windsor  with  Medium  confidence 
Confirm  as  SS  Windsor  with  Low  confidence 
Negate  Match  Without  providing  alternative 
Look  up  SS  Windsor  Database  Image 
Look  up  SS  Windsor  Satellite  image _ 

Negate  match  and  provide  alternative 


Choose  from  . . . 

Look  up  recently  received  satellite  imagery 
Look  up  recently  received  ATR  images 
Look  up  SS  Windsor  in  ship  images  database 
Look  up  SS  Windsor  in  satellite  images  database 
Browse  ship  images  database 
Browse  satellite  images  database 


Figure  7.  Example  of  a  Dynamic  Contextual  Decision  Tree  (from  Nehme,  2006),  that  allows  the  operator  to 
perform  a  confirmation  task  (top)  or  an  action  that  helps  them  perform  this  task  (bottom). 


ITT.  Addressing  Heterogeneity  and  Control  Allocation  in  the  Supervision  of  Complex  Unmanned 

Vehicle  Teams 

While  there  is  currently  increasing  interest  within  the  Navy  to  determine  system  architectures  that  would  allow 
one  person  or  a  small  team  of  people  the  ability  to  control  multiple,  heterogeneous  unmanned  vehicles,  there  has 
been  virtually  no  discussion  of  what  really  constitutes  heterogeneity,  particularly  from  a  human  cognitive  aspect.  For 
example,  heterogeneous  operations  could  include  vehicles  from  different  physical  domains  (i.e.,  UAV,USV,  UUV, 
etc.),  which  are  likely  to  vary  significantly  in  their  physical  operation.  Flowever,  multiple  vehicles  within  the  same 
domain — even  those  based  on  the  same  platform — can  also  exhibit  heterogeneity  with  respect  to  the  sensors  and 
weapon  packages  each  individual  vehicle  carries.  Vehicles  may  also  differ  across  a  wide  range  of  intelligent 
autonomy  capabilities,  requiring  interaction  that  ranges  from  remote  teleoperation  via  “stick  and  throttle”  control,  to 
“fire  and  forget”  monitoring  of  highly  autonomous  platforms  capable  of  mission  planning  and  execution  with  only 
high-level  objective-oriented  guidance  from  human  operators.  It  is  likely  that  all  three  of  these  types  of 
heterogeneity — operating  domain,  performance/payload  capabilities,  and  automation  capabilities — will  impact  the 
manner  in  which  a  human  operator  interacts  with  individual  vehicles  comprising  the  team. 

Beyond  differences  in  the  physical  and  operational  characteristics  of  individual  team  vehicles,  human  operators 
may  also  experience  heterogeneity  that  arises  from  the  assignment  of  these  vehicles  to  particular  mission  tasks,  as 
well  as  the  allocation  of  these  vehicles/tasks  across  one  or  more  operators — issues  related  more  closely  to  concepts 
of  operations  than  actual  differences  between  vehicles.  For  example,  task  heterogeneity  may  occur  if  an  operator  is 
responsible  for  supervising  two  unmanned  vehicles,  each  of  which  is  performing  separate,  distinct  mission  tasks. 
This  heterogeneity  would  require  the  operator  to  make  a  cognitive  switch  between  the  specific  requirements  and 
constraints  of  each  task  when  shifting  attention  between  vehicle  teams — something  that  would  not  be  required  if 
both  vehicles  were  engaged  in  the  same  task.  Similarly,  heterogeneity  of  location  may  occur  if  an  operator  is  tasked 
with  controlling  two  UUV  teams  operating  in  different  geographical  locations. 

Our  initial  cognitive  task  analysis  and  interface  design  effort,  described  in  the  previous  section,  focused  on  a 
simple  unmanned  vehicle  team  composed  of  four  identical  UUVs  performing  a  search  and  sentry  operation  and  one 
UAV  providing  communications  support.  Although  this  team  reflected  some  qualities  of  the  domain,  task,  and 
vehicle  heterogeneity  that  is  anticipated  for  future  unmanned  naval  missions  the  complexity  of  this  heterogeneity 
was  fairly  limited.  Even  so,  differences  between  the  tasks,  vehicle  capabilities,  and  operating  demands  across  this 
simple  multi-vehicle  team  had  a  great  impact  on  the  supervisory  responsibilities  of  the  human  operator,  as  well  as 
the  ways  in  which  the  operator  could  interact  with  the  unmanned  system  to  adapt  mission  behaviors.  This,  in  turn, 
affected  the  information  requirements  of  the  interactive  display  supporting  effective  supervisory  control,  as  well  as 
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the  particular  mechanisms  selected  to  convey  these  information  requirements  and  control  opportunities  to  the  human 
operator  for  mission  management.  Throughout  the  interface  design  effort,  it  became  increasingly  apparent  that 
significant  challenges  arise  when  creating  display  interfaces  capable  of  flexibly  supporting  highly  heterogeneous 
unmanned  vehicle  teams  and  mission  objectives.  This  is  because  the  cognitive  tasks  placed  upon  the  operator  during 
a  particular  operation  are  tightly  coupled  to  the  specific  vehicle  (or  vehicles)  performing  individual  mission  tasks, 
the  capabilities  of  the  automation  supporting  vehicle  behaviors,  and  the  allocation  of  tasks  and  vehicles  across  the 
supervisory  team.  Therefore,  it  cannot  be  assumed  that  the  previously  described  interface,  which  was  developed  to 
address  the  cognitive  challenges  facing  a  single  operator  supervising  five  particular  unmanned  vehicles  to  perform  a 
specific  operational  task,  would  appropriately  support  an  operator  controlling  a  different  team  of  heterogeneous 
vehicles.  Nor  would  the  interface  necessarily  support  the  same  team  of  vehicles  completing  different  operational 
tasks,  or  multiple  operators  collaboratively  sharing  vehicle  control. 

The  tight  coupling  between  mission  tasks,  vehicle  team  configuration,  and  operator  supervisory  control 
allocation  raises  numerous  questions  regarding  the  design  of  effective  displays  for  managing  complex  unmanned 
vehicle  teams.  For  example,  will  it  be  feasible  to  create  a  single,  generic  display  environment  that  is  capable  of 
flexibly  supporting  the  supervisory  control  of  any  unmanned  vehicle  in  performing  any  mission  task?  If  not,  will  it 
be  necessary  to  create  unique  interfaces  for  every  distinct  vehicle  domain  (e.g.  air  vehicles  vs.  underwater  vehicles)? 
For  every  distinct  vehicle  type  (e.g.  UAV  X  vs.  UAV  Y)?  For  every  vehicle/task  combination  (e.g.  a  UUV 
performing  surface  imaging,  vs.  the  same  UUV  performing  bottom  mapping)?  If  highly-specialized  supervisory 
interfaces  are  required,  will  it  be  possible  to  leverage  common  display  elements  and  interaction  techniques  in  a 
manner  that  allows  human  operators  to  easily  traverse  multiple  vehicle/task  configurations?  Could  such  displays  be 
designed  to  avoid  or  minimize  coordination  problems  and  the  cognitive  complexities  of  distributing  supervisory 
control  across  multiple,  highly  stove-piped  interaction  systems  during  complex  multi-vehicle  missions? 

These  questions  highlight  the  important  roles  in  determining  optimal  interface  design  that  are  played  by  (1) 
mission  heterogeneity,  and  (2)  alternative  strategies  for  vehicle/task  allocation  across  multi-member  supervisory 
teams.  In  the  remainder  of  this  section,  we  introduce  a  notional  scenario  that  we  have  developed  to  flexibly 
investigate  these  issues  of  heterogeneity  and  control  allocation  strategies  on  the  design  of  interfaces  for  supervising 
unmanned  vehicle  teams.  We  then  describe  the  methods  we  are  implementing  to  effectively  target  the  design  of 
future  supervisory  control  interfaces  by  augmenting  the  previously  described  hybrid  CTA  approach  with  modeling 
and  prototyping  efforts  that  analyze  where  these  heterogeneity  and  allocation  issues  may  present  the  greatest 
cognitive  challenge  to  the  operators  and  then  design  and  evaluate  interfaces  that  address  these  complexities. 

A.  Heterogeneity  of  Future  Unmanned  Missions 

To  support  a  more  detailed  investigation  of  these  issues,  the  mission  scenario  described  earlier  in  this  paper  was 
extended  to  better  reflect  multiple  forms  of  mission  heterogeneity  relevant  to  human  supervisory  control.  Although 
this  new  scenario  is  quite  complex  (and  far  reaching,  given  the  technological  capabilities  and  integration  of  existing 
unmanned  platforms),  it  reflects  numerous  attributes  of  heterogeneity  that  are  anticipated  within  futuristic  unmanned 
vehicle  teams  and  provides  a  rich  environment  for  exploring  the  cognitive  impact  of  heterogeneity  on  operator 
performance  and  interface  design  requirements. 

In  this  notional  scenario,  depicted  visually  in  Figure  8,  a  US  Navy  Littoral  Combat  Ship  (LCS)  is  stationed 
offshore  in  international  waters  and  tasked  with  preparing  to  insert  a  team  of  Special  Operations  Forces  deep  within 
an  enemy  harbor.  Prior  to  the  SOF  insertion,  the  LCS  launches  a  team  of  recoverable  UUVs  to  perform 
reconnaissance  on  the  planned  insertion  route  and  landing  site.  This  team  includes  one  submersible  (UUV1)  with  a 
mast-mounted  sensor  package  for  electro-optical  and  infrared  (EO/IR)  imaging  and  automated  classification  of 
surface  targets,  which  is  tasked  with  collecting  ISR  data  at  the  planned  insertion  site  ("Harbor  ISR"  task).  The  UUV 
team  also  includes  two  other  submersibles  (UUV2,  UUV3)  which  are  used  to  verify  and  secure  the  planned  insertion 
route  via  bottom-mapping  and  mine  countermeasure  operations  ("Bottom  Mapping/MCM"  task).  A  small,  rotor- 
based  UAV  (UAV1)  located  near  the  enemy  harbor  area  supports  communications  between  this  UUV  team  and  the 
LCS  platform  ("Comms"  task).  In  addition  to  the  UUV  team,  a  second  team  of  unmanned  vehicles  provides  anti¬ 
terrorism  and  force  protection  support  ("AT/FP"  task)  for  the  loitering  LCS.  This  team  includes  an  unmanned 
surface  vehicle  (USV1),  a  weaponized,  rigid-hulled  boat  tasked  with  patrolling  the  area  immediately  surrounding  the 
LCS  to  identify  and  intercept  small  vessels  that  may  threaten  the  ship.  The  USV  is  supported  by  another  small, 
rotor-based  UAV  (UAV2)  that  provides  additional  streaming  video  coverage  of  the  area,  as  well  as  targeting  support 
for  engaging  potential  threats. 
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1.  SOF  Insertion  Site 
and  planned  route 


2.  UUV  1  performs  clandestine 
Harbor  ISR  (EO  Imaging  of 
Surface  Targets) 


3.  UUV  2  and  UUV  3  perform 
bottom-mapping  and 
mine  countermeasure 
operations  along  planned 
insertion  route 


4.  UAV  1  supports 
COMMS  between 
UUV  team  and  LCS 


5.  USV  provides  SA  local  to 
LCS  and  supports  LCS  Anti- 
Terrorism  /  Force  Protection 
operations 


6.  UAV  2  provides  SA 
local  to  LCS  and 
supports  LCS  Anti- 
Terrorism  /  Force 
Protection  operations 


Figure  8.  Extended  operational  scenario  reflecting  significant  heterogeneity  in  a  notional  US  Navy 

unmanned  vehicle  mission. 

This  operational  scenario  highlights  four  distinct  tasks  within  the  mission  environment  (“Harbor  ISR,” 
“COMMS,”  “Bottom-Mapping/MCM”  and  “AT/FP”)  and  assigns  specific  unmanned  vehicles  to  each  of  these  tasks. 
The  scenario  does  not,  however,  specify  how  the  supervisory  control  of  these  vehicles  (or  tasks)  is  allocated 
amongst  a  team  of  one  or  more  human  operators.  Rather,  the  scenario  was  designed  to  flexibly  support  the  injection 
of  a  variety  of  notional  strategies  for  vehicle  control  allocation  across  the  overall  team,  each  resulting  in  different 
types  and  levels  of  mission  heterogeneity.  Three  of  these  strategies  for  control  allocation  are  shown  in  Figure  9  and 
described  below. 

Single  Operator  Allocation:  Under  this  operator  allocation  strategy,  control  over  all  vehicles  is  assigned  to  a 
single  operator  (Figure  9,  Configuration  1).  This  allocation  reduces  the  need  for  coordination  between  multiple 
human  operators,  but  leads  to  a  high  level  of  mission  heterogeneity.  In  this  case,  the  single  operator  is  tasked  with 
supervising  heterogeneous  vehicles  with  heterogeneous  capabilities,  performing  heterogeneous  tasks  in 
heterogeneous  domains  and  geographical  locations. 

Task-Based  Allocation:  Under  this  allocation  strategy  (Figure  9,  Configuration  2)  vehicle  control  is  distributed 
across  a  team  such  that  responsibility  for  a  particular  mission  task  (e.g.  AT/FP  for  the  LCS)  falls  to  a  single  human 
operator,  regardless  of  the  vehicles  used.  With  this  approach,  heterogeneity  of  task — and  likely  heterogeneity  of 
geographical  space — is  minimized.  However,  in  this  scenario,  preservation  of  task  homogeneity  leads  to 
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Table  9.  Effects  of  vehicle  and  task  allocation  strategies  on  mission  heterogeneity. 


Operator  A 
Operator  B 


Configuration  1 

Operator  A  responsible  for  all 
activities 


Configuration  2 

Vehicle  Allocation  by  Task  / 
Sector 


Configuration  3 

Vehicle  Allocation  by  Domain 


TASKS 


LCS  AT/FP 

•  LCS  threat 
surveillance 

•  Identification  of  tracks 
for  interrogation  / 
intercept 

•  Interception 


Harbor  ISR 

•  EO/IR  imaging  above 
the  surface  for 
insertion  recon 

•  Bottom  Mapping 


ISR  /  Comms 

•  Maintenance  of 
communications  relay 

•  MMTI,  EO/IR  imaging 


MCM/  Bottom  Mapping 

•  MCM  for  LCS  path 

•  Bottom  mapping  for 
insertion 


Region  A:  LCS  proximity, 
Harbor  OPAREA 


Tasks  A:  LCS  AT/FP, 
Mapping/MCM,  Harbor  ISR, 
Comms 


High  vehicle/domain 
heterogeneity  (A,  B) 


Domain  A:  Air/Water 
Domain  B:  Air/Water 
Region  A:  LCS  proximity 
Region  B:  Harbor  OPAREA 

Tasks  A:  LCS  AT/FP 
Tasks  B:  Mapping/MCM, 
Harbor  ISR,  Comms 


•  High  task/sector 
heterogeneity  (A,  B) 

•  High  vehicle  heterogeneity 
(A) 

•  Domain  A:  Water 

•  Domain  B:  Air 

•  Region  A:  LCS  proximity, 
Harbor  OPAREA 

•  Region  B:  LCS  proximity, 
Harbor  OPAREA 

•  Tasks  A:  LCS  AT/FP, 
Mapping/MCM,  Harbor  ISR 

•  Tasks  B:  LCS  AT/FP,  Harbor 
ISR,  Comms 


Notes 


High  task  complexity 
High  vehicle/domain 
heterogeneity 

Domain  A:  Air/Water 
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heterogeneity  of  vehicle  type  and  operating  domain.  For  example,  the  operator  tasked  with  protecting  the  LCS  must 
supervise  both  a  USV  and  a  UAV  to  perform  the  mission. 

Vehicle-Based  Allocation.  Under  this  allocation  strategy  (Figure  9,  Configuration  3)  vehicle  control  is 
distributed  across  a  pair  of  operators  based  on  vehicle  type  or  operating  domain,  regardless  of  the  tasks  being 
performed  by  that  vehicle.  In  this  scenario,  one  operator  is  responsible  for  supervising  all  water-based  vehicles  and 
the  second  operator  is  responsible  for  the  aerial  vehicles.  In  this  case,  vehicle  heterogeneity  is  greatly  reduced  for 
each  operator.  However,  task  and  location  heterogeneity  are  increased.  For  example,  although  the  second  operator  is 
only  supervising  UAVs,  each  vehicle  is  performing  a  separate  task  (e.g.,  AT/FP,  COMMS)  within  a  different  sector 
of  the  operational  area  (local  to  the  LCS  and  the  enemy  harbor,  respectively). 

Within  this  scenario,  it  is  clear  that  the  type  and  extent  of  mission  heterogeneity  experienced  by  mission 
operators  will  vary  greatly  with  the  strategy  for  asset  allocation  that  is  applied.  What  is  not  obvious,  however,  is  how 
these  different  allocation  strategies — and  the  resultant  heterogeneity  they  impart — will  impact  the  ability  of 
individual  operators  to  effectively  and  efficiently  supervise  the  unmanned  team,  nor  how  these  different  types  of 
heterogeneity  can  be  best  addressed  through  the  design  of  supervisory  control  interfaces.  For  example,  are  there 
particular  types  of  heterogeneity  that  will  be  more  detrimental  to  performance  than  others?  Will  preserving  task 
homogeneity  at  the  cost  of  vehicle  heterogeneity  lead  to  greater  operational  performance  than  when  vehicle 
homogeneity  is  preserved  at  the  cost  of  task  heterogeneity?  In  the  following  section,  we  describe  the  approach  we 
are  currently  using  to  investigate  these  concerns. 

B.  Evaluating  the  Effects  of  Heterogeneity  on  Mission  Performance 

The  mission  scenario  described  above  raises  numerous  concerns  regarding  the  effects  of  heterogeneity — a 
product  of  the  specific  operational  tasks,  unmanned  vehicles,  and  operator  allocation  for  a  given  mission — on  the 
efficiency  and  effectiveness  of  the  human  operator.  For  example:  Will  particular  types  of  heterogeneity  be  more 
detrimental  to  mission  performance  than  others?  How  can  the  cognitive  complexities  that  will  arise  from  different 
forms  of  heterogeneity  be  mitigated  through  interface  design?  In  light  of  these  concerns,  we  are  extending  our  initial 
interface  design  effort  to  address  comprehensive  interaction  solutions  for  the  supervisory  control  of  future 
heterogeneous  unmanned  vehicle  teams.  This  effort  includes  a  continuation  of  the  previously  described  hybrid  CTA 
and  design  process  to  account  for  the  expanded  mission  scenario,  which  incorporates  a  number  of  new  operational 
tasks  and  vehicles.  In  addition,  this  ongoing  work  will  also  employ:  (1)  a  simulation-based  modeling  effort  for  a 
predictive  analysis  of  how  various  types  and  levels  of  mission  heterogeneity  affect  human  and  overall  mission 
performance;  and  (2)  a  human-in-the-loop  evaluation  effort  to  rapidly  investigate  potential  interface  design 
approaches  for  addressing  heterogeneity  effects. 

Our  approach  begins  by  developing  a  better  understanding  of  how  heterogeneity  may  potentially  impact  mission 
performance  within  mixed-initiative  teams.  To  that  end,  we  are  constructing  a  framework  that  will  identify  different 
resource  allocation  strategies  involved  in  the  planning  and  re -planning  of  missions  that  require  teams  of  unmanned 
vehicles.  For  example,  one  can  consider  the  set  of  decisions  involved  in  allocating  each  vehicle  in  the  team  to 
particular  sub-tasks  of  the  overall  mission.  The  presence  of  heterogeneity  in  the  vehicle  team  will  affect  the 
alternatives  present  for  decision  making  related  to  such  assignment.  In  homogeneous  systems,  where  all  the  tasks 
and  vehicles  are  identical,  the  alternatives  for  vehicle-task  assignments  are  more  limited  than  in  the  case  where 
either  one  or  both  of  vehicles  and  tasks  are  heterogeneous.  An  important  implication  of  heterogeneity  in  unmanned 
vehicle  teams  is  therefore  an  increase  in  the  diversity  of  strategies  that  are  available  for  vehicle/operator/task 
resource  allocation. 

To  investigate  the  impact  of  heterogeneity  on  overall  mission  performance,  we  have  chosen  to  first  pursue 
discrete-event  simulations  of  the  previously  described  heterogeneous  scenario  so  that  the  effects  of  heterogeneity  in 
combination  with  mission  planning/re-planning  strategies  can  be  analyzed.  Our  simulation-based  approach  will 
allow  for  a  rapid  investigation  of  the  relative  contributions  of  the  many  types  and  levels  of  mission  heterogeneity 
highlighted  earlier.  The  results  from  the  simulations  will  be  used  as  part  of  an  effort  to  build  a  predictive  model  that 
can  estimate  mission  performance  as  vehicle-team  compositions  along  with  resource  allocation  strategies  are  varied. 
As  a  by-product  of  the  predictive  model,  planners  will  be  able  to  optimize  mission  performance  by  varying  one  or 
more  of  the  resource  allocation  strategies.  We  hope  to  use  this  model  to  provide  the  basis  for  decision  support  design 
that  recommends  to  the  mission  planners  and  re-planners  those  strategies  that  are  most  appropriate  for  maximizing 
mission  performance. 

Based  on  the  performance  insight  captured  by  this  modeling  effort,  we  will  apply  human  factors  design 
principles  (e.g.  Ref.  13,  Ref  14.),  to  transform  the  information  requirements  from  our  expanded  hybrid  CTA  into 
potential  interface  design  concepts.  These  interaction  design  concepts  will  then  be  instantiated  as  limited- 
functionality  software  interfaces,  using  a  variety  of  rapid  prototyping  methods.  We  will  use  these  prototypes  to 
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support  a  series  of  human-in-the-loop  evaluations  of  the  suitability  and  effectiveness  of  the  interface  design  concepts 
in  supporting  supervisory  control  of  heterogeneous  unmanned  vehicle  teams  in  a  simulated  mission  environment. 
The  mission  environment  itself  will  be  driven  by  an  experimental  testbed  that  we  are  currently  developing  to 
dynamically  link  the  prototype  display  interfaces  to  unmanned  vehicle  behaviors  and  evolving  environmental  states. 
When  possible,  this  testbed  will  leverage  existing  simulations  of  unmanned  vehicle  capabilities  and  behaviors,  such 
as  those  being  developed  under  the  Navy’s  Intelligent  Autonomy  program,  in  response  to  the  human  subject’s 
supervisory  control  actions.  Where  the  use  of  futuristic  or  notional  autonomous  capabilities  makes  this  simulation 
difficult  or  impossible,  the  testbed  will  support  light-weight  approaches  to  “faking”  these  capabilities  through  what 
are  commonly  referred  to  as  “Wizard-of-Oz”  experimental  techniques.  In  this  approach,  a  human  operator — unseen 
by  the  experimental  subject — serves  as  a  proxy  for  the  automation  system,  receiving  input  from  the  control  interface 
and  adapting  vehicle  behaviors  appropriately. 

This  human-centered  evaluation  process  will  target  specific  slices,  or  vignettes,  of  the  overall  mission  scenario, 
which  will  be  selected,  in  part,  based  on  predicted  operator  performances  generated  though  the  previous  modeling 
analysis.  For  each  of  these  vignettes,  human  subjects  will  use  the  prototype  display  interface  to  supervise  some 
subset  of  the  unmanned  vehicle  team  in  completing  mission  tasks  and  responding  to  dynamic  mission  events.  The 
subset  of  vehicles  under  the  operator’s  supervision  will  be  manipulated  to  reflect  the  range  of  previously  described 
vehicle  and  task  allocation  strategies.  Using  this  approach,  we  will  measure  a  variety  of  operator  performance 
metrics,  including  overall  mission  performance,  subjective  mental  workload,  and  response  time,  to  address  the 
effectiveness  with  which  the  prototype  display  interfaces  address  different  types  and  levels  of  mission  heterogeneity. 
Findings  from  this  evaluation  effort  will  be  used  to  guide  further  design  of  potential  supervisory  control  interface 
solutions,  as  well  as  to  further  evolve  our  operator  performance  models. 

IV.  Conclusions 

Future  military  unmanned  systems  will  be  highly  heterogeneous,  relying  upon  different  vehicles  from  a  range  of 
operating  domains  to  collaboratively  perform  complex  and  dynamic  mission  tasks.  While  these  systems  will  employ 
increasingly  sophisticated  automation,  the  human  operator  will  continue  to  play  a  fundamental  role  as  mission 
supervisor,  overseeing  vehicle  activities,  providing  contextual  insight  as  necessary,  and  responding  to  dynamic 
operational  environments  and  mission  goals.  Heterogeneity  in  vehicle  operating  domains,  performance 
characteristics,  levels  of  automation,  weapons  and  sensor  capabilities,  and  even  mission  tasks  will  all  contribute  to 
the  cognitive  load  placed  on  these  operators.  In  this  paper,  we  have  introduced  an  effort  to  address  these  effects  of 
heterogeneity  through  the  design  of  human-centered  information  displays.  We  first  presented  the  results  of  a 
cognitive  task  analysis  and  interface  design  effort  for  a  simple  unmanned  vehicle  team.  Following  this,  we  described 
the  issues  of  heterogeneity  and  coordination  of  command  across  multiple  vehicle  teams  as  two  key  concerns  in  the 
development  of  effective  and  efficient  interfaces  for  the  supervisory  control  of  unmanned  systems.  Finally,  we 
introduced  a  combined  simulation-based  modeling  and  human-in-the-loop  evaluation  approach  that  we  are  pursuing 
to  develop  supervisory  control  interfaces  that  flexibly  accommodate  alternative  command  and  control  philosophies 
for  heterogeneous  unmanned  vehicle  teams.  We  are  currently  applying  this  approach  in  the  development  and 
evaluation  of  control  interfaces  for  supporting  future  unmanned  naval  missions  within  the  littoral  combat 
environment. 
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